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Introduction 

What Is PIAAC? 

The Program for the International Assessment of Adult Competencies (PIAAC) is a cyclical, 
large-scale study of adult skills and life experiences focusing on education and employment. Nationally 
representative samples of adults between the ages of 16 and 65 are administered an assessment of 
literacy, numeracy, and problem solving in technology rich environments, as well as survey questions 
about their educational background, work history, the skills they use on the job and at home, their civic 
engagement, and sense of their health and well-being. The results are used to compare participating 
countries on the skills capacities of their workforce-aged adults and to learn more about relationships 
between educational background and employment and other outcomes. 

PIAAC is coordinated by the Organization for Economic Cooperation and Development (OECD) and 
developed by participating countries with the support of the OECD. PIAAC was first administered in 
201 1-12 in the United States and 23 other countries. 1 

What Is the U.S. PIAAC National Supplement? 

The U.S. PIAAC National Supplement is the second PIAAC data collection in the United States. Con- 
ducted from August 2013 through April 2014, the National Supplement’s household data collection 
surveyed 3,660 adults from three key U.S. subgroups of interest: 

• unemployed adults (age 16 to 65); 

• young adults (age 16 to 34); and 

• older adults (age 66 to 74). 

In addition to a household data collection, the National Supplement also surveyed 1,319 respondents 
from a target population defined as adult inmates (age 16 to 74) detained in federal and state prisons in 
the United States. 2 

The second round of U.S. PIAAC household data collection repeated the same procedures, instru- 
ments, and assessments that were used for the first round of data collection (called the “2012 PIAAC 
Main Study”) in the same sampled areas (or primary sampling units). However, while the first round 
collected data from a nationally representative sample of 5,010 adults, the second round collected data 
from 3,660 adults from only the three targeted (or oversampled) subgroups of interest. Together, the 
two rounds of U.S. PIAAC household data collection provide a nationally representative sample of 
8,670 noninstitutionalized adults in the United States between the ages of 16 and 74. It is important to 
note, however, that the second round of data is, by design, an oversample to supplement the first round 
of data collection and, hence, can only be used together with the first round of data. 

Why Was the National Supplement Conducted? 

The National Supplement was conducted for two reasons. First, augmenting the U.S. PIAAC sample 
permits more in-depth analyses of the cognitive and workplace skills of the U.S. population, in 
particular of the three key U.S. subgroups listed above. Second, the additional information on adults 
age 66 to 74 and incarcerated adults makes PIAAC data comparable with data collected by NCES in 
the 2003 National Assessment of Adult Literacy (NAAL). This in turn makes it possible to analyze 
change in adult skills over the decade between the two studies. 

1 In addition to the 23 countries that participated in PIAAC in 2012, 9 more countries (Chile, Greece, Indonesia, Israel, Lithuania, New Zealand, Singapore, 
Slovenia, and Turkey) took part in PIAAC in 2014. These additional countries will be added to the international PIAAC dataset in 2016 and thus are not 
included in the International Average shown in this report. These data will be available from the OECD and on the OECD International Data Explorer (IDE) 
at http:// www. oecd.org/ site/ piaac/ publicdataandanalysis.htm. 


2 Results from the prison data collection will be released in a separate report. 
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What Does PIAAC Measure? 


PIAAC is designed to assess adults in different countries over a broad range of abilities, from simple 
reading to complex problem-solving skills, and to collect information on individuals’ skill use and 
background. PIAAC defines four core competency domains of adult cognitive skills that are seen as key 
to facilitating the social and economic participation of adults in advanced economies: literacy, reading 
components, numeracy, and problem solving in technology-rich environments. All participating coun- 
tries and regions are required to assess the literacy and numeracy domains, but the reading components 
and problem solving in technology-rich environments domains are both optional. The United States 
assessed all four domains. For a list of the subject experts who contributed to the development of the 
PIAAC assessment, see appendix A. For a more detailed description of the four domains, see appendix B. 

Tasks developed for PIAAC’s four domains are authentic, culturally appropriate, and drawn from 
real-life situations that are expected to be important or relevant in different contexts. Tasks are intended 
to reflect adults’ daily lives across cultures, even if not every adult is necessarily familiar with every task. 
PIAAC is not designed to provide individual scores, but rather to measure how groups of adults perform 
on the domains. In order to be as efficient as possible with participants’ time, each respondent receives 
only a portion of the assessment items (see the Data Collection section of appendix C for more detail) . 

Literacy 

The primary goal of PIAAC’s literacy assessment is to measure everyday literacy, which is defined by the 
PIAAC framework as “understanding, evaluating, using and engaging with written text to participate in 
society, to achieve one’s goals and to develop one’s knowledge and potential ” (OECD 2012). 

Reading components 

The primary goal of the PIAAC reading components measure is to provide information about the 
literacy skills of adults at the lower end of the literacy spectrum — specifically, whether they have the 
foundational skills to develop the higher literacy and numeracy abilities necessary for functioning in 
society. The reading components assessment focuses on elements of reading that are comparable across 
the range of languages in the participating countries: reading vocabulary, sentence comprehension, 
and basic passage comprehension. Note that results for this domain are not shown in this report. 

Numeracy 

The primary goal of PIAAC’s numeracy assessment is to evaluate basic mathematical and computational 
skills that are considered fundamental for functioning in everyday work and social life. Numeracy in 
the PIAAC framework is defined as “the ability to access, use, interpret, and communicate mathematical 
information and ideas, to engage in and manage mathematical demands of a range of situations in adult 
life ” (OECD 2012). 

Problem solving in technology-rich environments 

PIAAC represents the first attempt to assess problem solving in technology-rich environments on a 
large scale and as a single dimension in an international context. PIAAC defines problem solving in 
technology-rich environments as “ using digital technology, communication tools, and networks to acquire and 
evaluate information, communicate with others, and perform practical tasks ” (OECD 2012). 

Skill use and the background questionnaire 

In addition to the skills assessment, PIAAC’s background questionnaire surveys adults about their edu- 
cational background; work history; their intrapersonal, interpersonal, and professional skills; and their 
use of those skills on the job and at home. 
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Reporting Results 

The purpose of this report is to present selected results from the first and second rounds of the U.S. PIAAC 
household data collection (PIAAC 20 12/20 14). 3 PIAAC results are reported in two ways: (1) as scale scores 
(estimated on a 0-500 scale) in the three domains of literacy, numeracy, and problem solving in technology- 
rich environments, 4 and (2) as percentages of adults reaching the proficiency levels established for each of 
these domains. 

PIAAC reports five proficiency levels for literacy and numeracy (Below level 1 , Level 1 , Level 2, Level 3, and 
Level 4/5) and four levels for problem solving in technology-rich environments (Below level 1, Level 1, Level 2, 
and Level 3). The OECD provides detailed descriptions of the types of skills that can be performed at each 
level. For example, adults at Level 1 in literacy can “read relatively short. . .texts to locate a single piece of 
information that is identical to or synonymous with the information given in the question or directive” and 
can “enter personal information onto a document” when “ [1] ittle, if any, competing information is 
present.” However, adults at Level 1 typically are not successful performing skills at the higher levels 
(e.g., “compare and contrast or reason about information requested” or “navigate within digital texts to 
access and identify information from various parts of a document,” both of which are Level 2 literacy 
skills). Appendix B provides the OECD’s detailed descriptions of these levels along with examples of 
assessment items at each level. 

This report follows OECD reporting conventions by combining the top two proficiency levels (Levels 
4 and 5) for the literacy and numeracy scales (OECD 2013). Across all countries, only 2 percent or less 
of adults performed at Level 5 in literacy and numeracy. 

This report also provides an international average for scale scores and proficiency levels for variables that are 
internationally comparable. The international averages in figures 3, 4, 5, 6, and 8 represent the averages for all 
participating countries and regions shown in this report and may differ slightly from the international averages 
reported in the 2012 NCES First Look (Goodman et al., 2013) (see below for details). 


How Does this First Look Report Differ from the 2012 PIAAC First Loolfl 


NCES’s 201 2 PIAAC First Look 
(Goodman et al., 2013) released in 
October 2013 presented data from 
the 2012 PIAAC Main Study. It 
focused on adult skills in the United 
States compared to the average of 
countries that participated in PIAAC 
in 2012 (the PIAAC international 
average), by major demographic 
characteristics. This First Look 
report presents the U.S. PIAAC 
2012/2014 data, which supports 
more detailed and precise esti- 
mates. 5 This First Look report 
updates some of the comparisons 


in the previous report, but also 
presents new results for unemployed 
adults, young adults, and older 
adults, which can now be analyzed 
because a larger sample is available 
to reliably estimate performance for 
these groups. 

The PIAAC international averages in 
the 2012 PIAAC First Look report 
were calculated using the U.S. 

2012 data and restricted-use data 
for Australia and Canada. However, 
data from Australia and Canada are 
not available because of national 


restrictions on the use of their data. 
Thus, the PIAAC international 
averages in figures 3, 4, 5, 6, and 
8 in this report were calculated (a) 
without Australian data, (b) with 
Canada’s publicly available PIAAC 
data, and (c) with the U.S. PIAAC 
2012/2014 data. Differences in the 
international averages calculated 
for the 2012 PIAAC First Look and 
those calculated for this report are 
very small but, on account of them, 
some estimates round differently. 


3 The “Selected Findings” in this report highlight results in some but not all figures presented in this report. 

4 Results from the reading components portion of the assessment are not reported in this report, but can be accessed from the IDE at 
http://www.oecd.org/site/piaac/publicdataandanalysis.htm. 

5 Besides the larger combined sample, the improved accuracy of estimates in this report is due in part to the revised population estimates based on the Census 2010 
data, which were unavailable when PIAAC 2012 went into the field. For more details, see appendix C - Changes in Population Totals from 2012 Main Study. 
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The percentage distribution of the population by key reporting variables is displayed in table 1 for all U.S. 
adults age 16-65, unemployed adults age 16-65, young adults age 16-34, and older adults age 66-74. 

Readers are cautioned not to draw causal inferences from results presented in this report. Many of the variables 
examined in this report may be related to one another, but the complex interactions and relationships among 
them have not been explored. The variables examined here are also just a few of the variables that can be 
examined in these data; they were selected to demonstrate the range of information available from the study. 
The release of this report is intended to encourage more in-depth analysis of the data using more sophisticated 
statistical methods. 

All statistically significant differences described in this report are significant at the .05 level. No statistical 
adjustments to account for multiple comparisons were used. Differences that are statistically significant 
are discussed using comparative terms such as “higher” and “lower.” Differences that are not statistically 
significant are either not discussed or referred to as “not measurably different” or “not statistically signifi- 
cant.” In this case, failure to find a difference as statistically significant does not necessarily mean that there 
was no difference. It could be that a real difference cannot be detected by the significance test because of a 
small sample size or an imprecise measurement in the sample. If the statistical test is significant, this means 
that there is convincing evidence (though no guarantee) of a real difference in the population. However, it 
is important to remember that statistically significant results do not necessarily identify those findings that 
have policy significance or practical importance. See appendix C for more information about statistical testing. 

This report provides findings for only a few select results, but more PLAAC results and resources are 
available from 

• The PIAAC Results Portal http:/ / nces.ed.gov/surveys/ piaac/ results/ makeselections.aspx . 

• The OECD’s PIAAC webpages http://www.oecd.org/site/piaac/ . 

• The International Data Explorer (IDE) http:/ / nces.ed.gov/ surveys/ piaac/ideuspiaac/ . 

A Brief History of PIAAC 

PIAAC’s development was based upon the pioneering work of national adult literacy assessments under- 
taken by the United States and Canada. In the United States, the first national assessment of literacy 
was the 1985 Young Adult Literacy Study (YALS), which was followed in 1992 by the National Adult 
Literacy Survey (NALS) and in 2003 by the National Assessment of Adult Literacy (NAAL). U.S. 
participation in international adult literacy assessments began in 1994 with the International Adult 
Literacy Survey (IALS), followed in 2003 by the Adult Literacy and Lifeskills Survey (ALL). 

Drawing on the knowledge and experience gained from these earlier national and international large- 
scale adult literacy assessments, PIAAC improves on these studies’ content frameworks, design, and 
methodologies to provide better measurement. PIAAC also extends the definitions of literacy and 
numeracy used by IALS and ALL; however, to ensure continuity with these previous studies and permit 
measurements of trend, PIAAC includes items from both IALS and ALL. 

Besides building on these earlier national and international studies, PIAAC includes various innova- 
tions that set it apart from these earlier studies. Lor example, PIAAC is the first large-scale assessment 
to be adaptive and administered on laptop computers to respondents in their homes. PIAAC provides 
more information about individuals with low levels of literacy by assessing reading component skills. To 
assess adults’ knowledge and skills for the digital age, PIAAC developed a new problem-solving domain 
with an emphasis on skills used in digital or technology-rich environments. 
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Selected Findings 

Overall summary for adults age 16-65 

In literacy, the U.S. average score (272) was not measurably different than the PIAAC international 
average score (273) (see figure 1-A). Compared with the PIAAC international average distribution of 
literacy skills, the United States had a larger percentage of adults performing at both the top and the 
bottom of the distribution (13 versus 12 percent at Level 4/5, and 18 versus 16 percent 6 at Level 1 
and below, see figure 2-A) . 

In numeracy and problem solving in technology-rich environments, the United States performed 
below the PIAAC international average. In numeracy, the U.S. average score was 12 points lower than the 
PIAAC international average score (257 versus 269, see figure 1-B), and in problem solving in technology-rich 
environments, the U.S. average score was 9 points lower than the international average (274 versus 283, 
see figure 1-C). Compared with the international average distributions for these skills, the United States 
had 

• a smaller percentage at the top (10 versus 12 percent at Level 4/5 in numeracy, and 5 versus 8 percent 
at Level 3 in problem solving in technology-rich environments, see figures 2-B and 2-C), and 

• a larger percentage at the bottom (28 versus 19 percent 6 in numeracy, and 64 versus 55 percent 6 in 
problem solving in technology- rich environments at Level 1 and below). 

Overall summary for U.S. adults age 16-74 by age 7 

The U.S. distribution of skills in each of the three domains for adults age 16-74, by age, suggested a 
relationship between age and performance. In literacy, the percentages of adults performing at the top 
proficiency level (4/5) were larger for adults age 25-34 and 35-44 than for adults at the other 10-year 
age intervals. In numeracy, the percentages of adults performing at the top proficiency level (4/5) were 
larger for those age 25-34 and 35-44 than those age 55-65 and 66 and older. For adults in the youngest 
age interval (16-24), the percentages at the top proficiency levels (4/5 for literacy and numeracy and 
3 for problem solving in technology-rich environments) were consistently smaller than for adults age 
25-34 (see figures 10-A, 10-B, and 10-C). 

Overall summary for U.S. adults age 16-65 by employment status 

In literacy, 15 percent of employed adults age 16-65 performed at the top proficiency level (4/5), while in 
numeracy 12 percent of employed adults reached this level. In both cases, the percentage of employed adults 
at the top proficiency level was larger than that of unemployed adults (7 percent in literacy and 4 percent in 
numeracy) and adults who were out of the labor force (9 percent in literacy and 6 percent in numeracy) 

(see figures 3-A and 3-B). 

Similarly, across all three domains, a larger percentage of adults age 16-65 who were unemployed and 
out of the labor force performed at or below Level 1 compared with adults who were employed (see 
figures 3-A, 3-B, and 3-C). 

Overall summary for unemployed adults age 16-65 

About 75 percent of unemployed U.S. adults age 16-65 had a high school credential or less education. 
Roughly a third of these adults performed at Level 1 or below in literacy (26 to 38 percent) and about 
half performed at Level 1 or below in numeracy (47 to 58 percent) (see figures 4-A and 4-B). 


6 Values for combined categories do not match figure due to rounding. 

7 These findings for the expanded U.S. adult population do not follow the sequence of figures in the report. The figures in the report are ordered to keep data in the same 
age ranges grouped together for ease of reference, whereas the Selected Findings on this page are ordered logically from the most general to more specific findings. 


The relationship between age and performance that was noted above among all U.S. adults differed for 
unemployed adults. Instead of the percentage of adults at the top proficiency level being smaller at older 
ages, among unemployed adults there was no measurable difference at the top proficiency level (4/ 5) in 
literacy and numeracy between those age 55-65 and age 25-34 (8 versus 6 percent in literacy and 5 versus 
4 percent in numeracy) (see figures 5 -A and 5-B). 

Comparing internationally, among unemployed adults age 16-65, larger percentages of both males and 
females in the United States performed at the bottom of the proficiency distribution (Level 1 or below) 
in numeracy and problem solving in technology-rich environments than is the case, on average, across 
the participating PIAAC countries (see figures 6-B and 6-C). 

Among unemployed U.S. adults age 16-65, larger percentages of unemployed White adults performed at 
the top proficiency level in all three domains (4/5 for literacy and numeracy and 3 for problem solving 
in technology-rich environments) than Black and Hispanic adults (12 versus 1 and 2 percent in literacy, 
7 versus 1 and 2 percent in numeracy, and 5 versus 1 percent each in problem solving in technology- 
rich environments) (see figures 7-A, 7-B, and 7-C). 

Overall summary for young adults age 16-34 

Among young adults age 16-34, the U.S. distribution of skills in each of the three domains by educa- 
tional attainment suggested a relationship between education and performance. In general, the higher 
the level of education completed, the larger the percentages of young adults at the top proficiency 
levels (4/ 5 for literacy and numeracy and 3 for problem solving in technology-rich environments) and 
the smaller the percentages at the bottom (Level 1 and below) (see figures 8-A, 8-B, and 8-C). 

Comparing internationally, among young adults age 1 6-34 whose highest level of education was high 
school or less, larger percentages in the United States performed at the bottom of the proficiency distri- 
bution (Level 1 or below) in all three domains than is the case, on average, across the participating PIAAC 
countries (see figures 8-A, 8-B, and 8-C). 

In the United States, smaller percentages of Black and Hispanic young adults age 16-34 performed at 
the top proficiency level (4/ 5) in literacy (4 and 5 percent) than their peers who reported their race/ 
ethnicity as White or Other (20 and 1 7 percent) (see figure 9-A) . 

Overall summary for older U.S. adults age 66-74 

In literacy and numeracy, there were no measurable differences in the percentage of older U.S. adults 
age 66-74 who performed at the highest proficiency level (4/ 5) between those who had a graduate or 
professional degree and those who had a bachelor’s degree as their highest educational attainment (19 
versus 19 percent in literacy and 18 versus 15 percent in numeracy) (see figures 1 1-A and 1 1-B). This 
differed from the pattern among young adults described above. 

Among older U.S. adults age 66-74, a greater percentage of those employed compared with those out of 
the labor force performed at the highest proficiency level (4/5) in numeracy (10 versus 4 percent) 

(see figure 12-B). 

Older U.S. adults age 66-74 who reported that their health status was “fair” had a larger percentage at the 
bottom of the proficiency distribution (Level 1 and below) in literacy and numeracy than their peers who 
reported that their health status was “good,” “very good,” or “excellent” (see figures 13-A and 13-B). 
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Tables and Figures 


Table 1. Percentage of U.S. adults in selected 

population groups, by selected characteristics: 2012 and 2014 1 


Characteristic 

All adults, 
age 16 to 65 

Unemployed adults, 
age 16 to 65 

Young adults, 
age 16 to 34 

Older adults, 
age 66 to 74 

Highest level of educational attainment 





Graduate or professional degree 

10 

5 

6 

13 

Bachelor's degree 

17 

10 

16 

14 

Associate's degree 

9 

8 

9 

6 

High school credential 

50 

54 

49 

53 

Below high school 

14 

23 

20 

14 

10-year age intervals 





16-24 

18 

33 

48 

t 

25-34 

20 

18 

52 

t 

35-44 

20 

17 

t 

t 

45-54 

21 

17 

t 

t 

55-65 

20 

14 

t 

t 

66-74 

t 

t 

t 

100 

Race/ethnicity 





White 

65 

52 

59 

81 

Black 

13 

21 

14 

8 

Hispanic 

15 

19 

19 

6 

Other 

7 

8 

8 

5 

Gender 





Male 

49 

47 

51 

47 

Female 

51 

53 

49 

53 

Highest level of parental education 





At least one parent attained college degree 

39 

35 

50 

19 

At least one parent attained high school degree 

44 

49 

39 

43 

Neither parent attained high school degree 

17 

16 

11 

39 

Born in the United States 





Yes 

85 

88 

88 

89 

No 

15 

12 

12 

11 

Self-reported health status 





Excellent 

23 

20 

29 

14 

Very good 

33 

29 

35 

29 

Good 

28 

32 

27 

33 

Fair 

11 

15 

8 

18 

Poor 

4 

3 

1 

6 

Diagnosed or identified as having a learning 
disability 





Yes 

8 

13 

10 

4 

No 

92 

87 

90 

96 


t Not applicable. 

1 United States data are the U.S. PIAAC 2012/2014 data. 

NOTE: The results shown in the table are based on the PIAAC literacy scale. Results from the PIAAC numeracy and problem solving in technology-rich environments scales are similar to the results from the 
literacy scale. Black includes African American, and Hispanic includes Latino. Race categories exclude Hispanic origin. Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic 
Cooperation and Development, PIAAC 2012. 
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Adults age 16 to 65 

Figure 1 -A. Average scores on the PIAAC literacy scale for adults age 16 to 65, by participating country and region: 2012 and 2014 1 
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Figure 1 -B. Average scores on the PIAAC numeracy scale for adults age 16 to 65, by participating country and region: 2012 and 2014 1 
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1 Data for all countries are from 2012, except for the United States which are the U.S. PIAAC 2012/2014 data. 

2 PIAAC 2012 international average based on all countries and regions that participated in PIAAC 2012 as reported in the 2012 NCES First Look (Goodman et al., 2013). Country- and region-specific results are available 
at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Countries and regions are listed in descending order determined by their unrounded average scores. Apparent differences between estimates may not be statistically significant. Some population groups did not 
have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic 
Cooperation and Development, PIAAC 2012. 


8 


Figure 1 -C. Average scores on the PIAAC problem solving in technology-rich environments scale for adults age 16 to 65, 
by participating country and region: 2012 and 201 4 1 
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1 Data for all countries are from 2012, except for the United States which are the U.S. PIAAC 2012/2014 data. 

2 PIAAC 2012 international average based on all countries and regions that participated in PIAAC 2012 as reported in the 2012 NCES First Look (Goodman et al., 2013). Country- and region-specific results are available at 
http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Countries and regions are listed in descending order determined by their unrounded average scores. No countries or regions scored significantly lower than the United States. Apparent differences between 
estimates may not be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. Not all countries chose to assess the problem solving in technology-rich 
environments domain. In countries that did, the percentage of respondents who completed the assessment varied widely from country to country but was less than the percentage that completed the literacy and 
numeracy domain because the assessment was only offered on computer. In the United States, approximately 15 percent of respondents chose not to take the assessment on computer or were unable to do so (see 
appendix C for more detailed information). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
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Figure 2-A. Percentage of adults age 1 6 to 65 at each level of proficiency on the PIAAC literacy scale: 201 2 and 2014 
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Figure 2-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale: 2012 and 2014 
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Figure 2-C. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale: 201 2 and 2014 
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1 United States data are the U.S. PIAAC 2012/2014 data. 

2 PIAAC 2012 international average based on all countries and regions that participated in PIAAC 2012 as reported in the 2012 First Look (NCES 2013-008). Country- and region-specific results are available at 
http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically significant. Some population groups did not have enough sample size to meet the 
minimum reporting standards. Not all countries chose to assess the problem solving in technology-rich environments domain. In countries that did, the percentage of respondents who completed the 
assessment varied widely from country to country but was less than the percentage that completed the literacy and numeracy domain because the assessment was only offered on computer. In the United 
States, approximately 15 percent of respondents chose not to take the assessment on computer or were unable to do so (see appendix C for more detailed information). 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic 
Cooperation and Development, PIAAC 2012. 


10 






Figure 3-A. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by employment status: 2012 and 201 4 1 
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Figure 3-B. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by employment status: 2012 and 201 4 1 
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Figure 3-C. Percentage of adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by employment status: 2012 and 2014 1 
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1 United States data are the U.S. PIAAC 2012/2014 data. PIAAC international average is calculated from the U.S. PIAAC 2012/2014 data and international data from 2012 for all other countries shown in this report. 
Country- and region-specific results are available at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Percentages of adults age 16 to 65 by employment status appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those adults who took the 
problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did not take the 
problem solving in technology-rich environments assessment (see appendix C for more detailed information). The unemployed comprise all persons above a specified age who during the reference period were in the 
following categories: without work, that is, were not in paid employment or self-employment during the reference period; currently available for work, that is, were available for paid employment or self-employment 
during the reference period; and seeking work, that is, had taken specific steps in a specified recent period to seek paid employment or self-employment. Out of the labor forces comprise all persons not classified as 
either employed or unemployed. Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically significant. Some population groups did not have enough sample 
size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic 
Cooperation and Development, PIAAC 2012. 
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Unemployed adults age 16 to 65 


Figure 4-A. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by highest level of educational attainment: 2012 and 2014 1 
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Figure 4-B. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by highest level of educational attainment: 2012 and 2014 1 
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Figure 4-C. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC problem solving 
in technology-rich environments scale, by highest level of educational attainment: 2012 and 201 4 1 
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Reporting standards not met. Sample size insufficient to permit a reliable estimate. 

' United States data are the U.S. PIAAC 2012/2014 data. PIAAC international average is calculated from the U.S. PIAAC 2012/2014 data and international data from 2012 for all other countries shown in this report. 

Country- and region-specific results are available at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

2 The PIAAC international average results for below level 1 of proficiency on the PIAAC literacy and numeracy scales are not available for unemployed adults age 16 to 65 whose highest level of education attainment is 
graduate or professional degree. 

NOTE: Percentages of unemployed adults age 16 to 65 by highest level of educational attainment appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only 
those adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did 
not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. Apparent differences between estimates may not 
be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation and 
Development, PIAAC 2012. 




























Figure 5-A. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by 10-year age intervals: 2012 and 2014 1 
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Figure 5-B. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by 10-year age intervals: 2012 and 2014 1 
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Figure 5-C. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC problem solving 
in technology-rich environments scale, by 10-year age intervals: 2012 and 201 4 1 
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1 United States data are the U.S. PIAAC 2012/2014 data. PIAAC international average is calculated from the U.S. PIAAC 2012/2014 data and international data from 2012 for all other countries shown in this report. 
Country- and region-specific results are available at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Percentages of unemployed adults age 16 to 65 by 10-year age intervals appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those adults 
who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did not take 
the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. Apparent differences between estimates may not be 
statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
























Figure 6-A. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by gender: 2012 and 2014 1 
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Figure 6-B. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by gender: 2012 and 2014 1 
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Figure 6-C. Percentage of unemployed adults age 16 to 65 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by gender: 2012 and 201 4 1 
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1 United States data are the U.S. PIAAC 2012/2014 data. PIAAC international average is calculated from the U.S. PIAAC 2012/2014 data and international data from 2012 for all other countries shown in this report. 
Country- and region-specific results are available at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Percentages of unemployed adults age 16 to 65 by gender appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those adults who took the 
problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did not take the problem 
solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically 
significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 


14 












Figure 7 -A. Percentage of unemployed U.S. adults age 16 to 65 at each level of proficiency on the PIAAC literacy scale, 
by race/ethnicity: 2012 and 2014 
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Figure 7-B. Percentage of unemployed U.S. adults age 16 to 65 at each level of proficiency on the PIAAC numeracy scale, 
by race/ethnicity: 2012 and 2014 
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Figure 7-C. Percentage of unemployed U.S. adults age 16 to 65 at each level of proficiency on the PIAAC problem solving 
in technology-rich environments scale, by race/ethnicity: 2012 and 2014 
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NOTE: United States data are the U.S. PIAAC 2012/2014 data. Percentages of unemployed U.S. adults age 16 to 65 by race/ethnicity appear in parentheses. The percentage distribution for problem solving in 
technology-rich environments scale includes only those adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment 
on computer or were unable to do so and therefore did not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Black includes African American, and 
Hispanic includes Latino. Race categories exclude Hispanic origin. Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically significant. Some population groups 
did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
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Young adults age 1 6 to 34 

Figure 8-A. Percentage of adults age 16 to 34 at each level of proficiency on the PIAAC literacy scale, by highest level of 
educational attainment: 2012 and 2014 1 
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Figure 8-B. Percentage of adults age 16 to 34 at each level of proficiency on the PIAAC numeracy scale, by highest level of 
educational attainment: 2012 and 2014 1 
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Figure 8-C. Percentage of adults age 16 to 34 at each level of proficiency on the PIAAC problem solving in technology-rich 
environments scale, by highest level of educational attainment: 2012 and 201 4 1 
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1 United States data are the U.S. PIAAC 2012/2014 data. PIAAC international average is calculated from the U.S. PIAAC 2012/2014 data and international data from 2012 for all other countries shown in this report. 
Country- and region-specific results are available at http://nces.ed.gov/surveys/piaac/results/makeselections.aspx. 

NOTE: Percentages of adults age 16 to 34 by highest level of educational attainment appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those 
adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did 
not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. Apparent differences between estimates may 
not be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
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Figure 9-A. Percentage of U.S. adults age 16 to 34 at each level of proficiency on the PIAAC literacy scale, 
by race/ethnicity: 2012 and 2014 
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Figure 9-B. Percentage of U.S. adults age 16 to 34 at each level of proficiency on the PIAAC numeracy scale, 
by race/ethnicity: 2012 and 2014 
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Figure 9-C. Percentage of U.S. adults age 16 to 34 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by race/ethnicity: 2012 and 2014 
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NOTE: United States data are the U.S. PIAAC 2012/2014 data. Percentages of U.S. adults age 16 to 34 by race/ethnicity appear in parentheses. The percentage distribution for problem solving in technology-rich 
environments scale includes only those adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or 
were unable to do so and therefore did not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Black includes African American, and Hispanic includes 
Latino. Race categories exclude Hispanic origin. Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically significant. Some population groups did not have 
enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
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Adults age 16 to 74 

Figure 1 0-A. Percentage of U.S. adults age 1 6 to 74 at each level of proficiency on the PIAAC literacy scale, 
by 10-year age intervals: 2012 and 2014 
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Figure 1 0-B. Percentage of U.S. adults age 1 6 to 74 at each level of proficiency on the PIAAC numeracy scale, 
by 10-year age intervals: 2012 and 2014 
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Figure 1 0-C. Percentage of U.S. adults age 1 6 to 74 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by 1 0-year age intervals: 2012 and 201 4 
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NOTE: United States data are the U.S. PIAAC 2012/2014 data. Percentages of U.S. adults age 16 to 74 by 10-year age intervals appear in parentheses. The percentage distribution for problem solving in technology-rich 
environments scale includes only those adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or 
were unable to do so and therefore did not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. 
Apparent differences between estimates may not be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. PIAAC 2012/2014; Organization for Economic Cooperation 
and Development, PIAAC 2012. 
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Older adults age 66 to 74 


Figure 11 -A. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC literacy scale, 
by highest level of educational attainment: 2014 
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Figure 1 1 -B. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC numeracy scale, 
by highest level of educational attainment: 2014 
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Figure 11 -C. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC problem solving in 


technology-rich environments scale, by highest level of educational attainment: 201 4 
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Reporting standards not met. Sample size insufficient to permit a reliable estimate. 

NOTE: Percentages of U.S. adults age 66 to 74 by highest level of educational attainment appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those 
adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did 
not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of rounding. Apparent differences between estimates may 
not be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. National Supplement, 2014. 
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Figure 1 2-A. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC literacy scale, 
by employment status: 2014 
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Figure 1 2-B. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC numeracy scale, 
by employment status: 2014 
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Figure 1 2-C. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC problem solving in 
technology-rich environments scale, by employment status: 2014 
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| Reporting standards not met. Sample size insufficient to permit a reliable estimate. 

NOTE: Percentages of U.S. adults age 66 to 74 by employment status appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes only those adults who took 
the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were unable to do so and therefore did not take the 
problem solving in technology-rich environments assessment (see appendix C for more detailed information). The unemployed comprise all persons above a specified age who during the reference period were in the 
following categories: without work, that is, were not in paid employment or self-employment during the reference period; currently available for work, that is, were available for paid employment or self-employment 
during the reference period; and seeking work, that is, had taken specific steps in a specified recent period to seek paid employment or self-employment. Out of the labor forces comprise all persons not classified as 
either employed or unemployed. Detail may not sum to totals because of rounding. Apparent differences between estimates may not be statistically significant. Some population groups did not have enough sample 
size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. National Supplement, 2014. 
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Figure 1 3-A. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC literacy scale, 
by self-reported health status: 2014 
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Figure 1 3-B. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC numeracy scale, 
by self-reported health status: 2014 
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Figure 1 3-C. Percentage of U.S. adults age 66 to 74 at each level of proficiency on the PIAAC problem solving 
in technology-rich environments scale, by self-reported health status: 2014 
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Reporting standards not met. Sample size insufficient to permit a reliable estimate. 

NOTE: Percentages of U.S. adults age 66 to 74 by self-reported health status appear in parentheses. The percentage distribution for problem solving in technology-rich environments scale includes 
only those adults who took the problem solving in technology-rich environments assessment. Approximately 15 percent of the U.S. sample chose not to take the assessment on computer or were 
unable to do so and therefore did not take the problem solving in technology-rich environments assessment (see appendix C for more detailed information). Detail may not sum to totals because of 
rounding. Apparent differences between estimates may not be statistically significant. Some population groups did not have enough sample size to meet the minimum reporting standards. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Program for the International Assessment of Adult Competencies (PIAAC), U.S. National Supplement, 2014. 
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More Information About PIAAC 

This report provides findings for only a few selected results. Readers may also 

• preview and print a selection of data on the performance of U.S. adults on the PIAAC assessment 
for various topics across all three domains at http://nces.ed.gov/surveys/piaac/index.asp ; 

• find more information about the international results at http://www.oecd.org/site/piaac/ ; 

• explore U.S. PIAAC data in the NCES International Data Explorer (IDE) at 
http: / / nces.ed. gov/ surveys/ international/ id e/ ; 

• access public-use data files that are available at http : // nces . ed -gov/ surveys/ piaac/ index. asp ; and 

• access restricted-use data files that are available to NCES Restricted-use Data Licensees. More 
information on licenses can be found at http://nces.ed.gov/pubsearch/licenses.asp . 
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Appendix A: Members of the PIAAC Subject 

Matter Expert Groups 

This appendix lists the members of the four expert groups responsible for developing the frameworks 
for the PIAAC content domains. 

Literacy Expert Group 

Stan Jones (Chair), Canada 

Egil Gabrielsen, Center for Reading Research, University of Stavanger, Norway 
Jan Hagston, Australia 

Pirjo Linnakyla, University of Jyvaskyla, Finland 

Hakima Megherbi, University of Paris, France 

John Sabatini, Educational Testing Service, United States of America 

Monika Troster, German Institute for Adult Education, Germany 

Eduardo Vidal-Abarca, Department of Psychology, University of Valencia, Spain 

Numeracy Expert Group 

Iddo Gal (Chair), University of Haifa, Israel 

Silvia Alatorre, National Pedagogical University, Mexico 

Sean Close, St. Patrick’s College, Ireland 

Jeff Evans, Middlesex University, United Kingdom 

Lene Johansen, Aalborg University, Denmark 

Terry Maguire, Institute of Technology Tallaght-Dublin, Ireland 

Myrna Manly, United States of America 

Dave Tout, Australian Council for Educational Research, Australia 

Problem Solving in Technology-Rich Environments 
Expert Group 

Jcan-Francois Rouet (Chair), National Center for Scientific Research (CNRS) and University 
of Poitiers, France 

Mirelle Betrancourt, University of Geneva, Switzerland 

M. Anne Britt, Northern Illinois University, United States of America 

Dr. Rainer Bromme, University of Muenster, Germany 

Arthur C. Graesser, University of Memphis, United States of America 

Jonna M. Kulikowich, Pennsylvania State University, United States of America 

Donald J. Leu, University of Connecticut, United States of America 

Naoki Ueno, Musashi Institute ofTechnology, Japan 

Herre van Oostendorp, Utrecht University, Netherlands 

Reading Components Expert Group 

John Sabatini, Educational Testing Service, United States of America 
Kelly M. Bruce, Educational Testing Service, United States of America 
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Appendix B: PIAAC Scales and Proficiency-Level 

Descriptions 

This appendix describes in more detail the PIAAC scales and the proficiency-level descriptions that 
accompany these scales. PIAAC proficiency results are also reported in terms of the percentages of 
adults performing at or “reaching” each of the proficiency levels. 

Overview 

PIAAC defines four core competency domains of adult cognitive skills that are seen as key to 
facilitating the social and economic participation of adults in advanced economies: 

• Literacy; 

• Reading components; 

• Numeracy; and 

• Problem solving in technology-rich environments. 

As described in appendix C, PIAAC is administered in either paper-and-pencil mode or via computer 
interface. Literacy and numeracy are offered in both paper-and-pencil and computer modes. Reading 
components, which are designed to provide information about the literacy skills of adults at the 
lower end of the literacy spectrum, are offered only in paper-and-pencil mode. Problem solving in 
technology-rich environments is administered via computer only. 

The OECD oversees the work of several teams of experts in the development of assessment 
frameworks in each of the domains (see appendix A). Assessment frameworks are available at http:// 
www.oecd.org/site/piaac/publications.htm . Information about the item development and proficiency 
level setting process will be included in a forthcoming PIAAC technical report from OECD. 

Literacy 

The PIAAC literacy framework expands on the definition of literacy previously used in IALS and ALL. 
PIAAC broadly defines literacy as “ understanding , evaluating, using and engaging with written text to 
participate in society, to achieve one’s goals and to develop one’s knowledge and potential ” (OECD 2012). 

The purpose of this expanded definition is to highlight the ranges of cognitive processes involved in 
literacy, focus on a more active role of individuals in society, and include various text types, both in 
print and electronic formats, in the measurement of literacy. 

PIAAC items include continuous texts (e.g., text in sentences and paragraphs), noncontinuous texts 
(e.g., schedules, graphs, and maps), and electronic texts (including hypertext or text in interactive 
environments, such as forms and blogs). Task activities are presented in home, work, and community 
contexts, addressing various purposes adults pursue in their everyday lives. 

Based on the PIAAC framework, literacy tasks include items in paper-and-pencil and computer-based 
delivery modes that cover a range of difficulties — low, middle, and high — to present a comprehensive 
picture of the level of adult literacy skills in each country or region. 
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Reading components 

The primary goal of the PIAAC reading components is to provide information about the literacy skills 
of adults at the lower end of the literacy spectrum — specifically, whether they have the foundational 
skills to develop the higher literacy and numeracy abilities necessary to function in society. 

The reading components assessment focuses on elements of reading that are comparable across the 
range of languages in the participating countries and regions: reading vocabulary, sentence comprehen- 
sion, and basic passage comprehension. 

• The reading vocabulary section asks participants to identify the best word that should be used to 
label different graphic illustrations. This task measures whether participants can identify common, 
concrete print words used in everyday adult interactions in the community, home, and workplace. 
It is not meant to determine the vocabulary knowledge (breadth or depth) of the participants. 

• The sentence comprehension section asks participants to identify whether sentences of varying 
grammatical/syntactic complexity make sense. This task measures whether participants can under- 
stand and correctly judge the accuracy of the content of sentences. 

• The basic passage comprehension section asks participants to make a choice between a correct 
and an incorrect word to complete a sentence within a passage. This task measures whether 
respondents comprehend text in context and can appropriately use words in ways that characterize 
fluency. 

The reading component portion of the assessment is optional for countries and regions participating in 
PIAAC. In countries and regions that adopt the reading components tasks, participants who decide not to 
take the computer-based assessment, and those who fail to pass the computer-administered information 
and communication technology (ICT) skills and literacy/numeracy “Core” items, are directed to the 
reading components tasks. (Additional information about the administration of the assessment and the 
“Core” items can be found in appendix C.) Data from the reading components portion of the assessment 
are not reported separately in this report, but can be accessed from the International Data Explorer 
(IDE) at http:/ /www.oecd.org/site/ piaac/ publicdataandanalysis.htm . 
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Exhibit B-1. Description of PIAAC proficiency levels on the literacy scale 


Proficiency levels 
and cut scores 
for literacy 

Literacy task descriptions 

Level 5 
(376 - 500) 

At this level, tasks may require the respondent to search for and integrate information across 
multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; 
or evaluate evidence-based arguments. Application and evaluation of logical and concep- 
tual models of ideas may be required to accomplish tasks. Evaluating reliability of eviden- 
tiary sources and selecting key information is frequently a requirement. Tasks often require 
respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use 
specialized background knowledge. 

Level 4 
(326 - 375) 

Tasks at this level often require respondents to perform multiple-step operations to integrate, 
interpret, or synthesize information from complex or lengthy continuous, non-continuous, 
mixed, or multiple type texts. Complex inferences and application of background knowledge 
may be needed to perform the task successfully. Many tasks require identifying and under- 
standing one or more specific, non-central idea(s) in the text in order to interpret or evalu- 
ate subtle evidence-claim or persuasive discourse relationships. Conditional information is 
frequently present in tasks at this level and must be taken into consideration by the respon- 
dent. Competing information is present and sometimes seemingly as prominent as correct 
information. 

Level 3 
(276 - 325) 

Texts at this level are often dense or lengthy, and include continuous, non-continuous, 
mixed, or multiple pages of text. Understanding text and rhetorical structures become more 
central to successfully completing tasks, especially navigating complex digital texts. Tasks 
require the respondent to identify, interpret, or evaluate one or more pieces of information, 
and often require varying levels of inference. Many tasks require the respondent to construct 
meaning across larger chunks of text or perform multi-step operations in order to identify 
and formulate responses. Often tasks also demand that the respondent disregard irrelevant 
or inappropriate content to answer accurately. Competing information is often present, but it 
is not more prominent than the correct information. 

Level 2 
(226 - 275) 

At this level, the medium of texts may be digital or printed, and texts may comprise con- 
tinuous, non-continuous, or mixed types. Tasks at this level require respondents to make 
matches between the text and information, and may require paraphrasing or low-level 
inferences. Some competing pieces of information may be present. Some tasks require the 
respondent to 

• cycle through or integrate two or more pieces of information based on criteria; 

• compare and contrast or reason about information requested in the question; or 

• navigate within digital texts to access and identify information from various parts of 
a document. 

Level 1 
(176-225) 

Most of the tasks at this level require the respondent to read relatively short digital or print 
continuous, non-continuous, or mixed texts to locate a single piece of information that is 
identical to or synonymous with the information given in the question or directive. Some 
tasks, such as those involving non-continuous texts, may require the respondent to enter 
personal information onto a document. Little, if any, competing information is present. Some 
tasks may require simple cycling through more than one piece of information. Knowledge 
and skill in recognizing basic vocabulary, determining the meaning of sentences, and read- 
ing paragraphs of text is expected. 

Below Level 1 
(0-175) 

The tasks at this level require the respondent to read brief texts on familiar topics to locate 
a single piece of specific information. There is seldom any competing information in the text 
and the requested information is identical in form to information in the question or directive. 
The respondent may be required to locate information in short continuous texts. However, in 
this case, the information can be located as if the text were non-continuous in format. Only 
basic vocabulary knowledge is required, and the reader is not required to understand the 
structure of sentences or paragraphs or make use of other text features. Tasks below level 1 
do not make use of any features specific to digital texts. 




Exhibit B-2. Examples of literacy items 


Items that exemplify the pertinent features of the proficiency levels in the domain of literacy are 
described below. In order to be consistent with the OECD international report, Levels 4 and 5 are 
combined in the figures in this report (Level 4/5). 

Level 4: Library search (Item ID: C323P002) 

Difficulty score: 348 

The stimulus displays results from a bibliographic search from a simulated library website. The test-taker is 
asked to identify a book suggesting that the claims made both for and against genetically modified foods are 
unreliable. He or she needs to read the title and the description of each book in each of the entries reporting 
the results of the bibliographic search in order to identify the correct book. Many pieces of distracting informa- 
tion are present. The information that the relevant book suggests that the claims for and against genetically 
modified foods are unreliable must be inferred from the truncated Internet search result stating that the author 
“describes how both sides in this hotly contested debate have manufactured propaganda, tried to dupe the 
public and... [text ends with ellipsis as shown]”. 

Level 3: Library search (Item ID: C323P003) 

Difficulty score: 289 

This task uses the same stimulus as the previous example. The test-taker is asked to identify the name of 
the author of a book called Ecomyth. To complete the task, the test-taker has to scroll through a list of bib- 
liographic entries and find the name of the author specified under the book title. In addition to scrolling, 
the test-taker must be able to access the second page where Ecomyth is located by either clicking the page 
number (2) or the word “next”. There is considerable irrelevant information in each entry to this particular 
task, which adds to the complexity of the task. 

Level 2: Lakeside fun run (Item ID: C322P002) 

Difficulty score: 240 

The stimulus is a simulated website containing information about the annual fun run/walk organized by 
the Lakeside community club. The test-taker is first directed to a page with several links, including “Contact 
Us” and “PAQs”. He or she is then asked to identify the link providing the phone number of organizers of 
the event. In order to answer this item correctly, the test-taker needs to click on the link “Contact Us”. This 
requires navigating through a digital text and some understanding of web conventions. While this task might 
be fairly simple for test-takers familiar with web-based texts, some respondents less familiar with web-based 
texts would need to make some inferences to identify the correct link. 
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Level 1: Generic medicine (Item ID: C309A321) 

Difficulty score: 219 

The stimulus is a short newspaper article entitled “Generic medicines: Not for the Swiss”. It has two 
paragraphs and a table in the middle displaying the market share of generic medicines in 14 European 
countries and the United States. The test-taker is asked to determine the number of countries in which 
the generic drug market accounts for 1 0 percent or more of total drug sales. The test-taker has to count 
the number of countries with a market share greater than 1 0 percent. The percentages are sorted in 
descending order to facilitate the search. The phrase “drug sales”, however, does not appear in the text; 
therefore, the test-taker needs to understand that “market share” is a synonym for “drug sales” in order 
to answer the question. 

Below Level 1: Election results (Item ID: C302BC02) 

Difficulty score: 1 62 

The stimulus consists of a short report of the results of a union election containing several brief paragraphs 
and a simple table identifying the three candidates in the election and the number of votes they received. 
The test-taker is asked to identify which candidate received the fewest votes. He or she needs to compare the 
number of votes that the three candidates received and identify the name of the candidate who received the 
fewest votes. The word “votes” appears in both the question and in the table and nowhere else in the text. 
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Numeracy 

The primary goal of PIAAC’s numeracy assessment is to evaluate basic mathematical and computational 
skills that are considered fundamental for functioning in everyday work and social life. Numeracy in 
the PIAAC framework is defined as “the ability to access, use, interpret, and communicate mathematical 
information and ideas, to engage in and manage mathematical demands of a range of situations in adult 
life ” (OECD 2012). 

The PIAAC numeracy domain is built on previous large-scale assessments of this domain, school- 
oriented assessments, and a review of requirements of workplace skills, adult learning, and mathematics 
and statistics education. The tasks that measure this domain involve managing a situation or solving a 
problem in a practical context — in home, work, or community settings. These tasks ask respondents 
to work with numbers, proportions, measurements, and statistical concepts, and then call for partici- 
pants to compute, interpret, and communicate the results and mathematical content. The situations 
and problems presented in these tasks involve objects or pictures, text, numbers, graphs, and 
technology-based displays. They also require basic mathematical skills in computation, proportions 
and percentages, an understanding of measurement concepts and procedures, and working with simple 
formulas. Respondents also encounter more complex items that require using models to predict future 
needs, and an understanding of basic statistical concepts and displays. 

In addition, PIAAC numeracy assessment items 

• are set in authentic and culturally appropriate contexts; 

• measure different levels of ability; and 

• use the standard measuring systems of the participating country or region. 

Numeracy tasks include items in paper- and-pencil and computer-based delivery modes that cover a range 
of difficulties — low, middle, and high — to present a comprehensive picture of the level of adult numeracy 
skills in each country or region. 
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Exhibit B-3. Description of PIAAC proficiency levels on the numeracy scale 


Proficiency levels 
and cut scores 
for numeracy 

Numeracy task descriptions 

Level 5 
(376 - 500) 

Tasks at this level require the respondent to understand complex representations and 
abstract and formal mathematical and statistical ideas, possibly embedded in complex 
texts. Respondents may have to integrate multiple types of mathematical information 
where considerable translation or interpretation is required; draw inferences; develop or 
work with mathematical arguments or models; and justify, evaluate and critically reflect 
upon solutions or choices. 

Level 4 
(326 - 375) 

Tasks at this level require the respondent to understand a broad range of mathematical 
information that may be complex, abstract or embedded in unfamiliar contexts. These 
tasks involve undertaking multiple steps and choosing relevant problem-solving strategies 
and processes. Tasks tend to require analysis and more complex reasoning about quanti- 
ties and data; statistics and chance; spatial relationships; and change, proportions and 
formulas. Tasks at this level may also require understanding arguments or communicating 
well-reasoned explanations for answers or choices. 

Level 3 
(276 - 325) 

Tasks at this level require the respondent to understand mathematical information that may 
be less explicit, embedded in contexts that are not always familiar and represented in more 
complex ways. Tasks require several steps and may involve the choice of problem-solving 
strategies and relevant processes. Tasks tend to require the application of number sense 
and spatial sense; recognizing and working with mathematical relationships, patterns, and 
proportions expressed in verbal or numerical form; and interpretation and basic analysis of 
data and statistics in texts, tables and graphs. 

Level 2 
(226 - 275) 

Tasks at this level require the respondent to identify and act on mathematical information 
and ideas embedded in a range of common contexts where the mathematical content is 
fairly explicit or visual with relatively few distractors. Tasks tend to require the application 
of two or more steps or processes involving calculation with whole numbers and common 
decimals, percents and fractions; simple measurement and spatial representation; estima- 
tion; and interpretation of relatively simple data and statistics in texts, tables and graphs. 

Level 1 
(176-225) 

Tasks at this level require the respondent to carry out basic mathematical processes in 
common, concrete contexts where the mathematical content is explicit with little text and 
minimal distractors. Tasks usually require one-step or simple processes involving count- 
ing, sorting, performing basic arithmetic operations, understanding simple percents such 
as 50%, and locating and identifying elements of simple or common graphical or spatial 
representations. 

Below Level 1 
(0-175) 

Tasks at this level require the respondents to carry out simple processes such as counting, 
sorting, performing basic arithmetic operations with whole numbers or money, or recogniz- 
ing common spatial representations in concrete, familiar contexts where the mathematical 
content is explicit with little or no text or distractors. 


Numeracy task descriptions 
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Exhibit B-4. Examples of numeracy items 


Items that exemplify the pertinent features of the proficiency levels in the domain of numeracy are 
described below. In order to be consistent with the OECD international report, Levels 4 and 5 are 
combined in the figures in this report (Level 4/5). No items mapped at Level 5 in numeracy. 

Level 4: Education level (Item ID: C632P001) 

Difficulty score: 354 

The stimulus for this item consists of two stacked-column bar graphs presenting the distribution of the 
Mexican population by years of schooling for men and women separately. The y-axis of each of the graphs is 
labeled “percentage” with 6 grid lines labeled “0%”, “20%”, “40%”, “60%”, “80%”, and “100%”. The x-axis 
is labeled “year” and data are presented for I960, 1970, 1990, 2000, and 2005. A legend identifies three 
categories of schooling: “more than 6 years of schooling”, “up to 6 years of schooling”, and “no schooling”. 
The test-taker is asked to approximate what percentage of men in Mexico had more than 6 years of schooling 
in 1970, choosing from a pull-down menu that has 10 response categories: “0-10%”, “10-20%”, and so on. 

Level 3: Package (Item ID: C657P001) 

Difficulty score: 315 

The stimulus for this item consists of an illustration of a box constructed from folded cardboard. The 
dimensions of the cardboard base are identified. The test-taker is asked to identify which plan best 
represents the assembled box out of four plans presented in the stimulus. 

Level 2: Logbook (Item ID: C613A520) 

Difficulty score: 250 

The stimulus for this item consists of a page from a motor vehicle logbook with columns for the date of 
the trip (start and finish), the purpose of the trip, the odometer reading (start and finish), the distance 
travelled, the date of entry and the driver’s name and signature. Lor the first date of travel (June 5), the 
column for the distance travelled is completed. The instructions inform the test-taker that “a salesman 
drives his own car and must keep a record of the miles he travels in a Motor Vehicle Log. When he travels, 
his employer pays him $0.35 per mile plus $40.00 per day for various costs such as meals.” The test-taker 
is asked to calculate how much he will be paid for the trip on June 5. 

Level 1: Candles (Item ID: C615A602) 

Difficulty score: 22 1 

The stimulus for this item consists of a photo of a box containing tea light candles. The packaging 
identifies the product (tea light candles), the number of candles in the box (105 candles) and its weight. 
While the packaging partially covers the top layer of candles, it can be seen that the candles are packed in 
five rows of seven candles each. The instructions inform the test-taker that there are 1 05 candles in a box 
and asks him or her to calculate how many layers of tea light candles are packed in the box. 

Below Level 1: Price tag (Item ID: C602A501) 

Difficulty score: 168 

The stimulus for this item consists of four supermarket price tags. These identify the product, the price per 
pound, the net weight, the date packed and the total price. The test-taker is asked to indicate the item that 
was packed first by simply comparing the dates on the price tags. 
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Problem solving in technology-rich environments 


PIAAC represents the first attempt to assess problem solving in technology-rich environments on a 
large scale and as a single dimension in an international context. PIAAC defines problem solving in 
technology-rich environments as “using digital technology, communication tools, and networks to acquire 
and evaluate information, communicate with others, and perform practical ’’(OECD 2012). 

Digital technology has revolutionized access to information and communication capabilities over the 
past two decades. In particular, the Internet has increased instantaneous access to large amounts of 
information and has expanded instant voice, text, and graphics capabilities across the globe. In order to 
effectively operate in these environments, it is necessary to have 

• knowledge of how various technological environments are structured (e.g., an understanding of 
the basics of the environment, including how to use command names, drop-down menus, naming 
protocols for files and folders, and links in a web page); and 

• the ability to interact effectively with digital information; understand electronic texts, images, 
graphics, and numerical data; and locate, evaluate, and critically judge the validity, accuracy, and 
appropriateness of the accessed information. 

These skills constitute the core aspects of the problem solving in technology-rich environments domain. 

Items in this domain present tasks of varying difficulty in simulated software applications using 
commands and functions commonly found in e-mail, web pages, and spreadsheets. These tasks range 
from purchasing particular goods or services online and finding interactive health information to 
managing personal information and business finances. 

PIAAC recognizes the diversity of digital technologies and the fact that they are evolving at a 
rapid pace, but due to implementation constraints, the first round of PIAAC was limited to using 
computers and simulated computer networks. The tasks assessing problem solving in technology-rich 
environments were only administered via computer and therefore only those taking the computerized 
assessment received a score in this domain. 
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Exhibit B-5. Description of PIAAC proficiency levels on the problem solving in 
technology-rich environments scale 


Proficiency levels 
and cut scores 
for problem 
solving in 
technology-rich 
environments 

Problem solving in technology-rich environments task descriptions 

Level 3 
(341 - 500) 

At this level, tasks typically require the use of both generic and more specific technology 
applications. Some navigation across pages and applications is required to solve the 
problem. The use of tools (e.g., a sort function) is required to make progress towards the 
solution. The task may involve multiple steps and operators. The goal of the problem 
may have to be defined by the respondent, and the criteria to be met may or may not 
be explicit. There are typically high monitoring demands. Unexpected outcomes and 
impasses are likely to occur. The task may require evaluating the relevance and reliability 
of information in order to discard distractors. Integration and inferential reasoning may be 
needed to a large extent. 

Level 2 
(291 - 340) 

At this level, tasks typically require the use of both generic and more specific technology 
applications. For instance, the respondent may have to make use of a novel online form. 
Some navigation across pages and applications is required to solve the problem. The use 
of tools (e.g., a sort function) can facilitate the resolution of the problem. The task may 
involve multiple steps and operators. The goal of the problem may have to be defined 
by the respondent, though the criteria to be met are explicit. There are higher monitoring 
demands. Some unexpected outcomes or impasses may appear. The task may require 
evaluating the relevance of a set of items to discard distractors. Some integration and 
inferential reasoning may be needed. 

Level 1 
(241 - 290) 

At this level, tasks typically require the use of widely available and familiar technology 
applications, such as e-mail software or a web browser. There is little or no navigation 
required to access the information or commands required to solve the problem. The 
problem may be solved regardless of the respondent’s awareness and use of specific tools 
and functions (e.g., a sort function). The tasks involve few steps and a minimal number of 
operators. At the cognitive level, the respondent can readily infer the goal from the task 
statement; problem resolution requires the respondent to apply explicit criteria; and there 
are few monitoring demands (e.g., the respondent does not have to check whether he or 
she has used the appropriate procedure or made progress towards the solution). Iden- 
tifying content and operators can be done through simple match. Only simple forms of 
reasoning, such as assigning items to categories, are required; there is no need to contrast 
or integrate information. 

Below Level 1 
(0 - 240) 

Tasks are based on well-defined problems involving the use of only one function within a 
generic interface to meet one explicit criterion without any categorical or inferential rea- 
soning, or transforming of information. Few steps are required and no sub-goal has to be 
generated. 


Problem solving in technology-rich environments task descriptions 
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Exhibit B-6. Examples of problem solving in technology-rich environments items 


Items that exemplify the pertinent features of the proficiency levels in the domain of problem solving in 
technology-rich environments are described below. 

Level 3: Meeting rooms (Item ID: U02) 

Difficulty score: 346 

This task involves managing requests to reserve a meeting room on a particular date using a reservation system. 
Upon discovering that one of the reservation requests cannot be accommodated, the test-taker has to send an 
e-mail message declining the request. Successfully completing the task involves taking into account multiple 
constraints (e.g., the number of rooms available and existing reservations). Impasses exist, as the initial con- 
straints generate a conflict (one of the demands for a room reservation cannot be satisfied). The impasse has 
to be resolved by initiating a new sub-goal, i.e., issuing a standard message to decline one of the requests. 
Two applications are present in the environment: an e-mail interface with a number of e-mails stored in 
an inbox containing the room reservation requests, and a web-based reservation tool that allows the user 
to assign rooms to meetings at certain times. The item requires the test-taker to “[u]se information from a 
novel web application and several e-mail messages, establish and apply criteria to solve a scheduling problem 
where an impasse must be resolved, and communicate the outcome.” The task involves multiple applications, 
a large number of steps, a built-in impasse, and the discovery and use of ad hoc commands in a novel environ- 
ment. The test-taker has to establish a plan and monitor its implementation in order to minimize the number 
of conflicts. In addition, the test-taker has to transfer information from one application (e-mail) to another 
(the room-reservation tool). 

Level 2: Club membership (Item ID: U19b) 

Difficulty score: 296 

This task involves responding to a request for information by locating information in a spreadsheet and 
e-mailing the requested information to the person who asked for it. The test-taker is presented with a 
word-processor page containing a request to identify members of a bike club who meet two conditions, 
and a spreadsheet containing 200 entries in which the relevant information can be found. The required 
information has to be extracted by using a sort function. The item requires the test- taker to “[o]rganize large 
amounts of information in a multiple-column spreadsheet using multiple explicit criteria and locate and 
mark relevant entries.” The task requires switching between two different applications and involves multiple 
steps and operators. It also requires some amount of monitoring. Making use of the available tools greatly 
facilitates identifying the relevant entries. 

Level 1: Party invitations (Item ID: U01A) 

Difficulty score: 286 

This task involves sorting e-mails into pre-existing folders. An e-mail interface is presented with five 
e-mails in an inbox. These e-mails are responses to a party invitation. The test-taker is asked to place 
the response e-mails into a pre-existing folder to keep track of who can and cannot attend a party. The 
item requires the test- taker to “[cjategorize a small number of messages in an e-mail application in existing 
folders according to a single criterion.” The task is performed in a single and familiar environment and 
the goal is explicitly stated in operational terms. Solving the problem requires a relatively small number 
of steps and the use of a restricted range of operators and does not demand a significant amount of 
monitoring across a large number of actions. 
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Appendix C: Methodology and Technical Notes 

This appendix describes the assessment design, sampling, data collection, weighting and variance 
estimation, scaling, and statistical testing procedures used to collect and analyze the data for this 
supplemental PIAAC report. The Main Study household data collection was conducted from 
August 25, 2011 through April 3, 2012, and the National Supplement household data collection 
occurred from August 26, 2013 through May 5, 2014. 

Assessment Design 

The PIAAC psychometric assessment design was complex because the assessment measured four 
domains — literacy, numeracy, reading components, and problem solving in technology-rich 
environments — across two modes of administration — paper-and-pencil and computer instruments. 

In summary, the PIAAC psychometric assessment design for the household sample provided for the 
following information: 

• reading component skills among lower performing adults in each participating country, as well as 
among those who reported not knowing how to use a computer; 

• population distributions in literacy, which could be linked to the International Adult Literacy 
Survey (IALS) and Adult Literacy and Lifeskills Survey (ALL); 

• population distributions in numeracy, which could be linked to ALL; 

• accurate estimates of population distributions and a baseline measure of problem solving in 
technology-rich environments for future estimation of trends over time; 

• insights into strategies and processes that adults used when they responded to the tasks on 
problem solving in technology-rich environments; 

• pairwise covariance estimates among the various measures, including the relationships between 
literacy and numeracy, literacy and reading components skills, literacy and problem solving in 
technology-rich environments, numeracy and reading components skills, and numeracy and 
problem solving in technology-rich environments; and 

• information that could be used to analyze the relationship between the measured competencies 
and the PIAAC behavioral measures and social/economic measures (from the responses to the 
background questions and job requirements approach module). 

PIAAC was designed as a computer-based assessment. Respondents who had little or no familiarity 
with computers, however, were directed to a paper-and-pencil version of the assessment that tested 
skills in the domains of literacy and numeracy only. Approximately 15 percent of the respondents in 
the Main Study and 23 percent in the National Supplement were directed to the paper-and-pencil 
path. Regardless of whether they took the assessment in the computer or paper-and-pencil format, all 
respondents first took a “Core” test to assess their capacity to undertake the full assessment. Those who 
were unsuccessful at the Core test were directed to the assessment of reading components. Those who 
succeeded at the Core test proceeded to the full assessment. 

The PIAAC assessment included an adaptive element that allowed for automatic scoring. Based on 
their performance at different points in the assessment, respondents taking the computer-based version 
were directed to different “testlets” that contained items of different average difficulty in the domains of 
literacy and numeracy. 
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Sampling 

For the 2012 PIAAC Main Study household sample, the target population consisted of non- 
institutionalized adults between the ages of 16 and 65 (inclusive). The assessment included a nationally 
representative probability sample of 9,468 housing units that resulted in 5,010 individual respondents. 
This household sample was selected on the basis of a four-stage, stratified area sample: (1) primary 
sampling units (PSUs) consisting of counties or groups of contiguous counties; (2) secondary sampling 
units (referred to as segments) consisting of area blocks; (3) housing units containing households; and 
(4) eligible persons within households. 

The purpose of the 2014 U.S. PIAAC National Supplement household sample was to provide additional 
samples of adults for the unemployed (age 16 to 65), two groups of young adults (age 16 to 24 and 25 
to 34), and a group of older adults (age 66 to 74). Given the low prevalence of unemployed adults in 
the population, a dual frame approach was implemented as an efficient method of sampling rare popu- 
lations. The dual frame approach consisted of an area sample and a list sample. An area sample of 9,579 
housing units was selected from the same PSUs and segments selected for the Main Study, after removing 
the housing units previously released. One or more persons from the National Supplement household 
sample target population was/were selected within a household. To obtain additional unemployed 
adults, five high unemployment Census tracts were identified within each of the sampled PSUs and one 
was randomly selected for the list sample. The United States Postal Service address list was purchased 
for each of the sampled tracts, and a sample of 6,956 households was taken from these lists. Within the 
sampled households, only those who were unemployed were eligible for selection. There were 3,660 
individuals that responded in the 2014 U.S. PIAAC National Supplement household sample which 
included the three subgroups of interest (unemployed adults, young adults, and older adults). 

For both the Main Study and the National Supplement household sample, person-level data were 
collected through a screener, a background questionnaire, and the assessment. The screener instrument 
was conducted using a computer-assisted personal interviewing system and collected information that 
included age and gender of all household members. It determined which household member or members 
was/were eligible for the study and selected the sample person(s). Of the 26,003 sampled housing units 
in the combined Main Study and National Supplement household sample, 3,500 were either vacant or 
not a housing unit, resulting in a sample of 22,503 households. A total of 9,460 households had at least 
one eligible adult and completed the screener (more than one adult per household could be selected to 
complete the questionnaire), which was used to select survey respondents. The final screener response 
rate was 84.7 percent weighted (note that all response rates referenced in this section were computed 
using base weights for the combined sample). 

Based on the screener data, 10,668 respondents age 16 to 74 were selected to complete the background 
questionnaire and the assessment; 8,488 actually completed the background questionnaire. Of the 
2,180 respondents who did not complete the background questionnaire, 182 were unable to do so 
because of a literacy-related barrier: either the inability to communicate in English or Spanish (the two 
languages in which the background questionnaire was administered), a reading or writing difficulty, 
or a mental disability. The final response rate for the background questionnaire — which included 
respondents who completed it and respondents who were unable to complete it because of a literacy- 
related barrier — was 80.9 percent weighted. 

Of the 8,488 adults age 16 to 74 who completed the background questionnaire, 8,341 completed the 
adult literacy assessment. An additional 26 were unable to complete the assessment for literacy-related 
reasons. The final response rate for the overall assessment — which included respondents who answered 
at least one question on each scale and the 26 respondents who were unable to do so because of a 
language problem or mental disability — was 98.8 percent weighted. 
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As described by PIAAC Technical Standard 4.3.3, a completed case contained at least the following: 

• responses to key background questions, including age, gender, highest level of schooling, and 
employment status; and 

• a completed Core instrument 1 (i.e., the interviewer asked the respondent all Core questions or 
the Core instrument was not completed for a literacy- related reason [e.g., because of a language 
difficulty] or because the respondent was unable to read or write in any of a country’s PIAAC 
official languages); or 

• responses to age and gender for literacy-related nonrespondents to the background questionnaire/ 
Job Requirements Approach. 

The overall weighted response rate for the household sample was 67.8 percent. 


Table C-l . Weighted response rate for the United States, by survey com 

ponent and sample: 2012 and 2014 



National Supplement 

Main Study 
and National 
Supplement 
Combined 
(percent) 

Component 

Main Study 
(percent) 

Area Sample 
(percent) 

List Sample 
(percent) 

Screener (household) 

86.5 

81.4 

84.8 

84.7 

Background questionnaire 

82.2 

78.1 

92.9 

80.9 

Assessment (without reading component) 

99.0 

98.5 

98.8 

98.8 

Overall rate (product of component response rates) 

70.3 

62.6 

77.9 

67.8 


For respondents who did not complete any tasks on any of the literacy scales, no information is 
available about their performance on the literacy items. Completely omitting these individuals from 
the analyses would have resulted in unknown biases in estimates of the literacy skills of the national 
population because refusals cannot be assumed to have occurred randomly. For respondents who 
answered the background questionnaire but refused to complete the assessment for reasons other than 
language issues or a mental disability, proficiency values were imputed based on the covariance 
information from those who completed the survey. 

The final household reporting sample (combined U.S. 2012 and 2014 sample) — including the literacy- 
related nonrespondents to the background questionnaire — consisted of 8,670 respondents. These 8,670 
respondents are the 8,490 respondents who completed the background questionnaire, plus the 180 respon- 
dents who were unable to complete the background questionnaire for literacy-related reasons. Of the 
roughly 8,490 respondents who completed the background questionnaire, about 1,450 were unemployed 
(age 16 to 65), 3,170 were not unemployed (age 16 to 34), 3,130 were not unemployed (age 35 to 65), 10 
had unknown employment status (age 16 to 65), and 730 had unknown employment status (age 66 to 74). 

The sample was subject to unit nonresponse from the screener, background questionnaire, assessment 
(including reading components), and item nonresponse to background questionnaire items. The 
screener and background questionnaire stages had unit response rates below 85 percent and thus 
required an analysis of the potential for nonresponse bias according to NCES statistical standards. 


1 For a full description of the Core instrument, see the section on Data Collection. 
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Nonresponse Bias 

The nonresponse bias analysis of the household sample revealed differences in the characteristics of 
respondents who participated in the background questionnaire compared with those who refused. In 
a bivariate unit-level analysis at the background questionnaire stage, estimated percentages for respon- 
dents were compared with those for the total eligible sample to identify any potential bias owing to 
nonresponse. Multivariate analyses were conducted to further explore the potential for nonresponse bias 
by identifying the domains with the most differential response rates. The three samples (Main Study, 
National Supplement area sample, and National Supplement list sample) were analyzed separately to 
inform the separate nonresponse weighting adjustments for each sample. 

For the Main Study, these analyses revealed that the subgroup with the lowest response rates for the 
background questionnaire had the following characteristics: (1) Hispanic, (2) age 26 and older with no 
children in the household, and (3) reside outside the Northeastern United States in areas with low levels 
of linguistic isolation (a low percentage who have some difficulty speaking English) and with unemploy- 
ment rates exceeding approximately 5 percent. For the National Supplement area sample, the lowest 
response rates to the background questionnaire were for persons with the following characteristics: (1) 
age 25 to 34 or older than 55, (2) sampled as not unemployed (age 16 to 34) or older (age 66 to 74), 

(3) no children in the household, (4) reside in the Northeastern United States in a census tract with an 
employment rate exceeding approximately 65 percent and in which more than approximately 2 percent 
of the population is foreign born. For the National Supplement list sample, the subgroup with the 
lowest background questionnaire response rate corresponded to the following: (1) female with no 
children in the household, (2) reside in a Metropolitan Statistical Area in the Western or Northeastern 
United States, and (3) reside in a census tract in which less than approximately 29 percent of the 
population has a high school education. 

In general, persons with children in the household were found to be more likely to participate, as were 
persons in areas with a high percentage of the population below 150 percent of poverty. However, 
the variables found to be significant in the multivariate analysis — those used to define areas with low 
response rates — were used in weighting adjustments in an effort to reduce bias. 

Data Collection 

Whenever possible, interviewers administered the background questionnaire and assessment in a private 
setting (e.g., home or library). Using the computerized interview and assessment software provided by 
the PIAAC Consortium, 2 the interviewer read the background questionnaire questions from a laptop 
and entered all responses directly into the laptop. Skip patterns and follow-up probes for contradictory 
or out-of-range responses were programmed into the interview software. At the completion of the back- 
ground questionnaire, the participant was administered the computer-based Core or the paper-and- 
pencil based Core if the participant could not or would not use the computer. Upon the completion 
and scoring of the Core tasks, the respondent was routed to the computer-based assessment (CBA), the 
paper-based assessment (PBA) of literacy and numeracy, or the paper-based reading components. The 
background questionnaire and the assessment took approximately two hours to complete; however, the 
time varied by the respondent. The number of assessment items also varied based on the respondents’ 
performance on the Core and the adaptive routing implemented in the automated portion of the 
assessment. 


2 The PIAAC Consortium includes the following organizations: Educational Testing Service (ETS), Westat, cApStAn, the Research Centre for Education and the 
Labor Market (ROA), gesis-ZUMA Centre for Survey Research, German Institute for International Education Research (DIPF), and the Data Processing Centre 
of the International Association for the Evaluation of Educational Achievement (IEA). In addition to these organizations, PIAAC is aided by numerous national 
contracting partners. 
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Progress through the assessment was controlled by the computer based on the respondents 
performance on various components of the assessment. The PIAAC assessment was composed 
of the following components: 

• The Core consisted of three modules: the CBA Core Stage 1, the CBA Core Stage 2, and the 
PBA Core. 

o The CBA Core Stage 1 included six tasks and was designed to determine if the participant had 
the basic set of information and communication technology (ICT) skills needed to complete the 
computer-based assessment. To pass the CBA Core Stage 1 , the participant needed to correctly 
answer at least three of the first five tasks, plus the sixth task (highlighting text) . CBA Core 
Stage 1 questions were automatically scored by the computer, and a participant who passed 
the CBA Core Stage 1 continued on to the CBA Core Stage 2. A participant who did not pass 
the CBA Core Stage 1 was routed to the PBA Core. 

o The CBA Core Stage 2 included six tasks that measured basic literacy and numeracy skills 
necessary to undertake the assessment. CBA Core Stage 2 questions were automatically scored 
by the computer, and a participant who passed the CBA Core Stage 2 continued on to the 
computer-based assessment. A participant who did not pass the CBA Core Stage 2 was routed 
directly to the paper-based reading components section. 

o The PBA Core consisted of eight tasks and measured basic literacy and numeracy skills 
necessary to undertake the assessment. PBA Core questions were interviewer scored and 
entered into the computer to determine if the participant passed the PBA Core. A participant 
who passed the PBA Core continued on to the paper-based assessment of literacy and numer- 
acy and then to the paper-based reading components section. A participant who did not pass 
the PBA Core was routed directly to the reading components section. 

• The assessment was administered in CBA and PBA modes. 

o The CBA consisted of three “testlets” of tasks at Stage 1 (9 items) and four “testlets” at Stage 2 
( 1 1 items) . Each respondent completed two testlets that included items from two of the three 
domains. 

o The PBA consisted of two paper-based assessment booklets, one contained literacy items and 
one contained numeracy items. Each booklet contained 20 items for the participant to com- 
plete and each participant completed only one booklet type. 

• The reading components were completed by a participant after completing the literacy or numer- 
acy booklet. Reading components were also completed by a respondent who failed the CBA Core 
Stage 2 or the PBA Core. 
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Problem Solving in Technology-Rich Environments: U.S. Sample 

The PIAAC assessment design was developed to route respondents to the most appropriate delivery 
mode as a means to help assure the most reliable, valid, and comparable assessment of skills. The 
computer-based assessment (CBA) was chosen for those demonstrating information and communication 
technology (ICT) skills, while the remaining respondents received the paper-based assessment (PBA). 
The scores for respondents who had no computer experience, failed the ICT skills test, or refused the 
CBA did not contribute to the estimation of the item parameters for the problem solving in technology- 
rich environments (PS-TRE) domain. The design of the PIAAC assessment contained only literacy and 
numeracy in the PBA because the problem solving in technology-rich environments assessment, by 
definition, was suitable only for respondents familiar with ICT environments. Exhibit C-l illustrates 
the different stages of the assessment administration and the different pathways that respondents could 
follow based on their responses to questions on ICT use in the background questionnaire (BQ). For 
each pathway, the weighted percentages of U.S. respondents who followed that pathway are shown. For 
example, the percentage of U.S. respondents who said they had some computer experience and were 
thus routed to the CBA was 89.2 percent, but the percentage of all U.S. respondents who were routed 
to the CBA and eventually received the literacy and numeracy assessment was 19.2 percent while the 
percentage who received literacy and PS-TRE was 6.8 percent. 

Exhibit C-1. PIAAC Main Study and National Supplement yield: 2012 and 2014 



Weighting and Variance Estimation 

A dual-frame, complex sample design was used to select assessment respondents. The properties of 
a sample selected through a complex design could be very different from those of a simple random 
sample in which every individual in the target population has an equal chance of selection and in which 
the observations from different sampled individuals can be considered statistically independent of one 
another. Therefore, the properties of the sample for the complex data collection design were taken into 
account during the analysis of the data. 

One way of addressing the properties of the sample design was by using sampling weights to composite 
the Main Study and National Supplement household samples and account for the fact that the prob- 












abilities of selection were not identical for all respondents. For the household sample, the sampling 
weights were further adjusted for nonresponse to the screener and background questionnaire, extreme 
weights were trimmed, and weights for all respondents were calibrated to the U.S. Census Bureau’s 
2012 American Community Survey population totals for those age 16 to 74. Since literacy-related 
nonrespondents to the screener, the background questionnaire, and the assessment are similar in 
prohciency, the weights of the literacy-related nonresponse cases were not adjusted during the screener- 
level nonresponse adjustment. Instead, the background questionnaire weights for the background 
questionnaire and assessment literacy-related cases were adjusted to account for the literacy-related 
screener nonrespondents. This adjustment was necessary primarily to allow the literacy-related 
background questionnaire and assessment nonrespondents to represent the literacy-related screener 
nonrespondents in the calibration procedure. 

All population and subpopulation characteristics based on the PIAAC data used sampling weights 
in their estimation. The statistics presented in this report are estimates of group and subgroup perfor- 
mance based on a sample of respondents, rather than the values that could be calculated if every person 
in the nation answered every question on the instrument. Therefore, it is important to have measures 
of the degree of uncertainty of the estimates. Accordingly, in addition to providing estimates of 
percentages of respondents and their average scale scores, this report provides information about 
the uncertainty of each statistic in the form of standard errors on the U.S. PIAAC website at 
http:// nces.ed.gov/ surveys/ piaac/ results/ summary, aspx . 

Because the assessment used clustered sampling, conventional formulas for estimating sampling variability 
(e.g., standard errors) that assume simple random sampling and hence independence of observations would 
have been inappropriate for this report. For this reason, the PIAAC assessment used a paired jackknife 
replication approach (sometimes referred to as JK2) to estimate standard errors (Rust and Rao 1996). 

Changes in Population Totals from 2012 Main Study 

For the 2012 data collection, weights for all respondents were calibrated to the U.S. Census Bureau’s 
2010 American Community Survey population totals for those age 16 to 65. The 2010 American 
Community Survey population totals were derived from 2000 U.S. Census projections because the full 
2010 U.S. Census population results were not yet available. Once the 2010 U.S. Census population 
results were finalized, the U.S. Census refreshed its entire time series of estimates going back to the 
previous census each year using the most current data and methodology. One result of this refresh is a 
shift in the proportion of the population with more education. A comparison of the population totals 
used to calibrate the 2012 Main Study data with those used to calibrate the composite 2012 and 2014 
data reveals that the percentage of the U.S. population age 16 to 65 with college experience (some 
college or a college degree) increased by 3 to 4 percent and the percentage of the population age 16 
to 65 with less than a high school diploma decreased 4 percent. This change has no effect on PIAAC’s 
measurement of skills in the United States, but it does mean that the proportions of the population 
with higher skills have been found to be larger than previously estimated for the 2012 Main Study. 

Scaling 

Information on scaling in the PIAAC assessment can be found on the OECD PIAAC website at 
http://www.oecd.org/site/piaac/ . 

Statistical Testing 

The statistical comparisons in this report were based on the t statistic. Statistical significance was deter- 
mined by calculating a t value for the difference between a pair of means or proportions, and comparing 
this value with published tables of values at a certain level of significance, called the alpha level. The alpha 
level is an a priori statement of the probability of inferring that a difference exists when, in fact, it does 
not. Findings from r- tests are reported based on a statistical significance (or alpha level) set at .05, without 
adjustments for multiple comparisons. 
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